Estimating Frequency Moments of Streams

We will develop algorithms that can approximate Fk by making one pass of the stream and using a small amount of memory o(n+m). Frequency moments have a number of applications. F0 represents the number of distinct elements in the streams (which the FM-sketch from last class estimates using O(log n) space. F1 is the number of elements in the stream m. F2 is used in database optimization engines to estimate self join size. Consider the query, “return all pairs of individuals that are in the same location”. Such a query has cardinality equal to ∑ im 2 i /2, where mi is the number of individuals at a location. Depending on the estimated size of the query, the database can decide (without actually evaluating the answer) which query answering strategy is best suited. F2 is also used to measure the information in a stream. In general, Fk represents the degree of skew in the data. If Fk/F0 is large, then there are some values in the domain that repeat more frequently than the rest. Estimating the skew in the data also helps when deciding how to partition data in a distributed system.